Pandas Profiling Report

Dataset statistics

Number of variables	11
Number of observations	1013
Missing cells	25
Missing cells (%)	0.2%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	177.9 KiB
Average record size in memory	179.8 B

Variable types

Numeric	7
Categorical	3
Boolean	1

Warnings

`answered_correctly` is highly correlated with `content_type_id`	High correlation
`user_answer` is highly correlated with `content_type_id`	High correlation
`content_type_id` is highly correlated with `answered_correctly` and 1 other fields	High correlation
`prior_question_elapsed_time` has 24 (2.4%) missing values	Missing
`df_index` is uniformly distributed	Uniform
`row_id` is uniformly distributed	Uniform
`user_id` is uniformly distributed	Uniform
`df_index` has unique values	Unique
`row_id` has unique values	Unique
`timestamp` has unique values	Unique
`user_id` has unique values	Unique

Reproduction

Analysis started	2021-01-11 06:02:10.695355
Analysis finished	2021-01-11 06:02:18.594259
Duration	7.9 seconds
Software version	pandas-profiling v2.10.0
Download configuration	config.yaml

df_index
Real number (ℝ_≥0)

UNIFORM
UNIQUE

Distinct	1013
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	50600000
Minimum	0
Maximum	101200000
Zeros	1
Zeros (%)	0.1%
Memory size	8.0 KiB

Quantile statistics

Minimum	0
5-th percentile	5060000
Q1	25300000
median	50600000
Q3	75900000
95-th percentile	96140000
Maximum	101200000
Range	101200000
Interquartile range (IQR)	50600000

Descriptive statistics

Standard deviation	29257221.33
Coefficient of variation (CV)	0.5782059552
Kurtosis	-1.2
Mean	50600000
Median Absolute Deviation (MAD)	25300000
Skewness	0
Sum	5.12578 × 10¹⁰
Variance	8.55985 × 10¹⁴
Monotocity	Strictly increasing

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	1	0.1%
79200000	1	0.1%
14700000	1	0.1%
99700000	1	0.1%
40300000	1	0.1%
101200000	1	0.1%
65400000	1	0.1%
67200000	1	0.1%
32900000	1	0.1%
67700000	1	0.1%
Other values (1003)	1003	99.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	0.1%
100000	1	0.1%
200000	1	0.1%
300000	1	0.1%
400000	1	0.1%

Value	Count	Frequency (%)
101200000	1	0.1%
101100000	1	0.1%
101000000	1	0.1%
100900000	1	0.1%
100800000	1	0.1%

row_id
Real number (ℝ_≥0)

UNIFORM
UNIQUE

Distinct	1013
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	50600000
Minimum	0
Maximum	101200000
Zeros	1
Zeros (%)	0.1%
Memory size	8.0 KiB

Quantile statistics

Minimum	0
5-th percentile	5060000
Q1	25300000
median	50600000
Q3	75900000
95-th percentile	96140000
Maximum	101200000
Range	101200000
Interquartile range (IQR)	50600000

Descriptive statistics

Standard deviation	29257221.33
Coefficient of variation (CV)	0.5782059552
Kurtosis	-1.2
Mean	50600000
Median Absolute Deviation (MAD)	25300000
Skewness	0
Sum	5.12578 × 10¹⁰
Variance	8.55985 × 10¹⁴
Monotocity	Strictly increasing

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	1	0.1%
79200000	1	0.1%
14700000	1	0.1%
99700000	1	0.1%
40300000	1	0.1%
101200000	1	0.1%
65400000	1	0.1%
67200000	1	0.1%
32900000	1	0.1%
67700000	1	0.1%
Other values (1003)	1003	99.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	0.1%
100000	1	0.1%
200000	1	0.1%
300000	1	0.1%
400000	1	0.1%

Value	Count	Frequency (%)
101200000	1	0.1%
101100000	1	0.1%
101000000	1	0.1%
100900000	1	0.1%
100800000	1	0.1%

timestamp
Real number (ℝ_≥0)

UNIQUE

Distinct	1013
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	7597207750
Minimum	0
Maximum	6.995499156 × 10¹⁰
Zeros	1
Zeros (%)	0.1%
Memory size	8.0 KiB

Quantile statistics

Minimum	0
5-th percentile	606192
Q1	606228481
median	2890290907
Q3	9940892027
95-th percentile	3.187169706 × 10¹⁰
Maximum	6.995499156 × 10¹⁰
Range	6.995499156 × 10¹⁰
Interquartile range (IQR)	9334663546

Descriptive statistics

Standard deviation	1.103981536 × 10¹⁰
Coefficient of variation (CV)	1.45314117
Kurtosis	6.546620077
Mean	7597207750
Median Absolute Deviation (MAD)	2790943396
Skewness	2.360220435
Sum	7.695971451 × 10¹²
Variance	1.218775232 × 10²⁰
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	1	0.1%
3631362798	1	0.1%
6.176883233 × 10¹⁰	1	0.1%
3255399758	1	0.1%
969006415	1	0.1%
7772069200	1	0.1%
2.407519573 × 10¹⁰	1	0.1%
72006999	1	0.1%
1473101144	1	0.1%
6552298843	1	0.1%
Other values (1003)	1003	99.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	0.1%
9988	1	0.1%
20772	1	0.1%
24225	1	0.1%
28835	1	0.1%

Value	Count	Frequency (%)
6.995499156 × 10¹⁰	1	0.1%
6.80589582 × 10¹⁰	1	0.1%
6.706899271 × 10¹⁰	1	0.1%
6.444268417 × 10¹⁰	1	0.1%
6.264806611 × 10¹⁰	1	0.1%

user_id
Real number (ℝ_≥0)

UNIFORM
UNIQUE

Distinct	1013
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1076410247
Minimum	115
Maximum	2146925942
Zeros	0
Zeros (%)	0.0%
Memory size	8.0 KiB

Quantile statistics

Minimum	115
5-th percentile	108335132.4
Q1	540770973
median	1071637713
Q3	1615528747
95-th percentile	2039444869
Maximum	2146925942
Range	2146925827
Interquartile range (IQR)	1074757774

Descriptive statistics

Standard deviation	620448906.8
Coefficient of variation (CV)	0.5764056114
Kurtosis	-1.199824749
Mean	1076410247
Median Absolute Deviation (MAD)	537511510
Skewness	0.0009360069152
Sum	1.090403581 × 10¹²
Variance	3.849568459 × 10¹⁷
Monotocity	Strictly increasing

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
1732722689	1	0.1%
77694321	1	0.1%
1146987861	1	0.1%
2003697688	1	0.1%
927090007	1	0.1%
1946583025	1	0.1%
1819112934	1	0.1%
1426500957	1	0.1%
1059622238	1	0.1%
1820753016	1	0.1%
Other values (1003)	1003	99.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
115	1	0.1%
2078569	1	0.1%
4022163	1	0.1%
5615405	1	0.1%
7783335	1	0.1%

Value	Count	Frequency (%)
2146925942	1	0.1%
2145211573	1	0.1%
2143675324	1	0.1%
2141804691	1	0.1%
2139715142	1	0.1%

content_id
Real number (ℝ_≥0)

Distinct	905
Distinct (%)	89.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	5168.663376
Minimum	5
Maximum	32312
Zeros	0
Zeros (%)	0.0%
Memory size	8.0 KiB

Quantile statistics

Minimum	5
5-th percentile	396.6
Q1	2065
median	4727
Q3	7219
95-th percentile	10792.6
Maximum	32312
Range	32307
Interquartile range (IQR)	5154

Descriptive statistics

Standard deviation	3840.365747
Coefficient of variation (CV)	0.7430094529
Kurtosis	6.668281786
Mean	5168.663376
Median Absolute Deviation (MAD)	2627
Skewness	1.554806342
Sum	5235856
Variance	14748409.07
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
1278	5	0.5%
7216	5	0.5%
7217	4	0.4%
6173	4	0.4%
7490	3	0.3%
4120	3	0.3%
2948	3	0.3%
1223	3	0.3%
4492	3	0.3%
2063	3	0.3%
Other values (895)	977	96.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
5	1	0.1%
10	1	0.1%
12	1	0.1%
26	1	0.1%
36	1	0.1%

Value	Count	Frequency (%)
32312	1	0.1%
29579	1	0.1%
29544	1	0.1%
26711	1	0.1%
26335	1	0.1%

content_type_id
Categorical

HIGH CORRELATION

Distinct	2
Distinct (%)	0.2%
Missing	0
Missing (%)	0.0%
Memory size	1.2 KiB

0	990
1	23

Length

Max length	1
Median length	1
Mean length	1
Min length	1

Characters and Unicode

Total characters	1013
Distinct characters	2
Distinct categories	1 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	0
2nd row	0
3rd row	0
4th row	0
5th row	0

Value	Count	Frequency (%)
0	990	97.7%
1	23	2.3%

Histogram of lengths of the category

Value	Count	Frequency (%)
0	990	97.7%
1	23	2.3%

Most occurring characters

Value	Count	Frequency (%)
0	990	97.7%
1	23	2.3%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	1013	100.0%

Most frequent character per category

Value	Count	Frequency (%)
0	990	97.7%
1	23	2.3%

Most occurring scripts

Value	Count	Frequency (%)
Common	1013	100.0%

Most frequent character per script

Value	Count	Frequency (%)
0	990	97.7%
1	23	2.3%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	1013	100.0%

Most frequent character per block

Value	Count	Frequency (%)
0	990	97.7%
1	23	2.3%

task_container_id
Real number (ℝ_≥0)

Distinct	721
Distinct (%)	71.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	914.7897335
Minimum	1
Maximum	9458
Zeros	0
Zeros (%)	0.0%
Memory size	8.0 KiB

Quantile statistics

Minimum	1
5-th percentile	13
Q1	122
median	411
Q3	1044
95-th percentile	3712
Maximum	9458
Range	9457
Interquartile range (IQR)	922

Descriptive statistics

Standard deviation	1392.047226
Coefficient of variation (CV)	1.521712777
Kurtosis	11.31826223
Mean	914.7897335
Median Absolute Deviation (MAD)	351
Skewness	3.080239446
Sum	926682
Variance	1937795.478
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
15	14	1.4%
5	8	0.8%
22	7	0.7%
1	5	0.5%
3	5	0.5%
211	5	0.5%
167	5	0.5%
9	5	0.5%
10	5	0.5%
122	5	0.5%
Other values (711)	949	93.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	5	0.5%
2	1	0.1%
3	5	0.5%
4	4	0.4%
5	8	0.8%

Value	Count	Frequency (%)
9458	1	0.1%
9276	1	0.1%
9148	1	0.1%
8891	1	0.1%
8769	1	0.1%

user_answer
Categorical

HIGH CORRELATION

Distinct	5
Distinct (%)	0.5%
Missing	0
Missing (%)	0.0%
Memory size	1.3 KiB

3	271
0	262
1	261
2	196
-1	23

Length

Max length	2
Median length	1
Mean length	1.022704837
Min length	1

Characters and Unicode

Total characters	1036
Distinct characters	5
Distinct categories	2 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	3
2nd row	3
3rd row	0
4th row	3
5th row	0

Value	Count	Frequency (%)
3	271	26.8%
0	262	25.9%
1	261	25.8%
2	196	19.3%
-1	23	2.3%

Histogram of lengths of the category

Value	Count	Frequency (%)
1	284	28.0%
3	271	26.8%
0	262	25.9%
2	196	19.3%

Most occurring characters

Value	Count	Frequency (%)
1	284	27.4%
3	271	26.2%
0	262	25.3%
2	196	18.9%
-	23	2.2%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	1013	97.8%
Dash Punctuation	23	2.2%

Most frequent character per category

Value	Count	Frequency (%)
1	284	28.0%
3	271	26.8%
0	262	25.9%
2	196	19.3%

Value	Count	Frequency (%)
-	23	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	1036	100.0%

Most frequent character per script

Value	Count	Frequency (%)
1	284	27.4%
3	271	26.2%
0	262	25.3%
2	196	18.9%
-	23	2.2%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	1036	100.0%

Most frequent character per block

Value	Count	Frequency (%)
1	284	27.4%
3	271	26.2%
0	262	25.3%
2	196	18.9%
-	23	2.2%

answered_correctly
Categorical

HIGH CORRELATION

Distinct	3
Distinct (%)	0.3%
Missing	0
Missing (%)	0.0%
Memory size	1.2 KiB

1	660
0	330
-1	23

Length

Max length	2
Median length	1
Mean length	1.022704837
Min length	1

Characters and Unicode

Total characters	1036
Distinct characters	3
Distinct categories	2 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	1
2nd row	0
3rd row	1
4th row	0
5th row	1

Value	Count	Frequency (%)
1	660	65.2%
0	330	32.6%
-1	23	2.3%

Histogram of lengths of the category

Value	Count	Frequency (%)
1	683	67.4%
0	330	32.6%

Most occurring characters

Value	Count	Frequency (%)
1	683	65.9%
0	330	31.9%
-	23	2.2%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	1013	97.8%
Dash Punctuation	23	2.2%

Most frequent character per category

Value	Count	Frequency (%)
1	683	67.4%
0	330	32.6%

Value	Count	Frequency (%)
-	23	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	1036	100.0%

Most frequent character per script

Value	Count	Frequency (%)
1	683	65.9%
0	330	31.9%
-	23	2.2%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	1036	100.0%

Most frequent character per block

Value	Count	Frequency (%)
1	683	65.9%
0	330	31.9%
-	23	2.2%

prior_question_elapsed_time
Real number (ℝ_≥0)

MISSING

Distinct	223
Distinct (%)	22.5%
Missing	24
Missing (%)	2.4%
Infinite	0
Infinite (%)	0.0%

Mean	25862.31143
Minimum	0
Maximum	300000
Zeros	1
Zeros (%)	0.1%
Memory size	8.0 KiB

Quantile statistics

Minimum	0
5-th percentile	7000
Q1	15000
median	21000
Q3	30000
95-th percentile	60000
Maximum	300000
Range	300000
Interquartile range (IQR)	15000

Descriptive statistics

Standard deviation	21953.08386
Coefficient of variation (CV)	0.8488446178
Kurtosis	57.56096619
Mean	25862.31143
Median Absolute Deviation (MAD)	7000
Skewness	5.740463196
Sum	25577826
Variance	481937890.9
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
17000	49	4.8%
16000	47	4.6%
19000	41	4.0%
22000	38	3.8%
18000	37	3.7%
14000	34	3.4%
15000	33	3.3%
21000	30	3.0%
20000	29	2.9%
23000	28	2.8%
Other values (213)	623	61.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1	0.1%
333	1	0.1%
1250	1	0.1%
1333	2	0.2%
1666	1	0.1%

Value	Count	Frequency (%)
300000	1	0.1%
293000	1	0.1%
243400	1	0.1%
141000	1	0.1%
138000	1	0.1%

prior_question_had_explanation
Boolean

Distinct	2
Distinct (%)	0.2%
Missing	1
Missing (%)	0.1%
Memory size	1.2 KiB

True	899
False	113
(Missing)	1

Common Values
Chart

Value	Count	Frequency (%)
True	899	88.7%
False	113	11.2%
(Missing)	1	0.1%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Cramér's V (φc)

Cramér's V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér's V have been proved to be biased, even for large samples. We use a bias-corrected measure that has been proposed by Bergsma in 2013 that can be found here.

A simple visualization of nullity by column.

Nullity matrix is a data-dense display which lets you quickly visually pick out patterns in data completion.

The correlation heatmap measures nullity correlation: how strongly the presence or absence of one variable affects the presence of another.

The dendrogram allows you to more fully correlate variable completion, revealing trends deeper than the pairwise ones visible in the correlation heatmap.

First rows

	df_index	row_id	timestamp	user_id	content_id	task_container_id	user_answer	answered_correctly	prior_question_elapsed_time	prior_question_had_explanation
0	0	0	0	115	5692	1	3	1	NaN	NaN
1	100000	100000	153893981	2078569	5673	280	3	0	56000.0	True
2	200000	200000	186936199	4022163	1296	74	0	1	20000.0	True
3	300000	300000	577329	5615405	4120	9	3	0	23000.0	False
4	400000	400000	27552108270	7783335	1481	358	0	1	5666.0	True
5	500000	500000	1782558219	9678259	8313	571	3	0	16000.0	True
6	600000	600000	124095	11945677	2064	4	3	0	18000.0	False
7	700000	700000	18739740135	14442037	12	1031	1	1	18000.0	True
8	800000	800000	374628459	16202138	5402	30	0	1	33000.0	True
9	900000	900000	23452298654	18789336	8144	955	0	0	4250.0	True

Last rows

	df_index	row_id	timestamp	user_id	content_id	task_container_id	user_answer	answered_correctly	prior_question_elapsed_time	prior_question_had_explanation
1003	100300000	100300000	2047497893	2129118709	6873	211	2	0	24250.0	True
1004	100400000	100400000	22662923203	2131647088	9950	1247	3	1	24000.0	True
1005	100500000	100500000	8495176626	2133622536	1118	1910	0	0	14000.0	True
1006	100600000	100600000	384366	2136087039	7984	19	0	1	19000.0	False
1007	100700000	100700000	2861070155	2137979150	5117	606	2	1	10000.0	True
1008	100800000	100800000	62504905	2139715142	594	45	3	1	16000.0	True
1009	100900000	100900000	5342702918	2141804691	7234	4018	3	1	45500.0	True
1010	101000000	101000000	31269876280	2143675324	1119	776	1	0	14000.0	True
1011	101100000	101100000	4759608402	2145211573	8568	1152	3	1	10000.0	True
1012	101200000	101200000	9269436738	2146925942	11859	163	1	1	26000.0	True

Overview

Variables

Most occurring characters

Most occurring categories

Most frequent character per category

Most occurring scripts

Most frequent character per script

Most occurring blocks

Most frequent character per block

Most occurring characters

Most occurring categories

Most frequent character per category

Most occurring scripts

Most frequent character per script

Most occurring blocks

Most frequent character per block

Most occurring characters

Most occurring categories

Most frequent character per category

Most occurring scripts

Most frequent character per script

Most occurring blocks

Most frequent character per block

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Cramér's V (φc)

Missing values

Sample

First rows

Last rows